feature: Inferentia Neuron support for HuggingFace #2976

jeniyat · 2022-03-08T03:04:34Z

Issue #, if available:

Description of changes:

Added the necessary changes to incorporate the inf-neuron/hf image into the huggigface framework

Testing done:

Locally tested the feature to extract image string with the following code, where the output matched with the released DLC image

import sagemaker, boto3
sagemaker.image_uris.retrieve(framework="huggingface",
    region=boto3.Session().region_name,
    version="4.12.3",py_version="py37",
    base_framework_version="pytorch1.9.1",
    inference_tool="neuron",
    image_scope="inference")
'763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference-neuron:1.9.1-transformers4.12.3-neuron-py37-sdk1.17.1-ubuntu18.04'

from sagemaker.huggingface import HuggingFaceModel
import sagemaker
import boto3
sess = sagemaker.Session()

role = sagemaker.get_execution_role()

huggingface_model = HuggingFaceModel(
    model_data='s3://sagemaker-us-west-2-267274314323/hf-sagemaker-inf/model.tar.gz',
	transformers_version='4.12',
	pytorch_version='1.9',
	py_version='py37',
	role=role, 
    sagemaker_session=sess
)

huggingface_model._is_compiled_model = True
huggingface_model.prepare_container_def("ml.inf.xlarge")

{'Image': '763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference-neuron:1.9.1-transformers4.12.3-neuron-py37-sdk1.17.1-ubuntu18.04', 'Environment': {'SAGEMAKER_PROGRAM': '', 'SAGEMAKER_SUBMIT_DIRECTORY': '', 'SAGEMAKER_CONTAINER_LOG_LEVEL': '20', 'SAGEMAKER_REGION': 'us-west-2'}, 'ModelDataUrl': 's3://hf-sagemaker-inference/inferentia/model.tar.gz'}

import sagemaker, boto3
from sagemaker.huggingface import HuggingFaceModel
sess = sagemaker.Session()
role = sagemaker.get_execution_role()
huggingface_model = HuggingFaceModel(
    model_data='s3://sagemaker-us-west-2-267274314323/hf-sagemaker-inf/model.tar.gz',                
    transformers_version='4.12.3',
    pytorch_version='1.9.1',
    py_version='py37',
    role=role, 
    sagemaker_session=sess)
huggingface_model._is_compiled_model = True
predictor = huggingface_model.deploy(initial_instance_count=1, instance_type="ml.inf1.xlarge")
huggingface_model.image_uri


'763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference-neuron:1.9.1-transformers4.12.3-neuron-py37-sdk1.17.1-ubuntu18.04'

Tested the changes in this PR locally, where it passed all the tests (attached the local output below)

JT:sagemaker-python-sdk jeniyat$ ./.githooks/pre-push
GLOB sdist-make: /Users/jeniyat/Desktop/HuggingFace/source_repo/sagemaker-python-sdk/setup.py
✔ OK black-check in 10.19 seconds
✔ OK twine in 12.616 seconds
✔ OK pylint in 22.618 seconds
✔ OK docstyle in 24.769 seconds
✔ OK flake8 in 41.238 seconds
__________________________________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________________________________
  flake8: commands succeeded
  pylint: commands succeeded
  docstyle: commands succeeded
  black-check: commands succeeded
  twine: commands succeeded
  congratulations :)
=================== flake8,pylint,docstyle,black-check,twine execution time ===================
44 seconds

GLOB sdist-make: /Users/jeniyat/Desktop/HuggingFace/source_repo/sagemaker-python-sdk/setup.py
✔ OK doc8 in 7.752 seconds
✔ OK sphinx in 4 minutes, 57.083 seconds
__________________________________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________________________________
  sphinx: commands succeeded
  doc8: commands succeeded
  congratulations :)
=================== sphinx,doc8 execution time ===================
4 minutes and 59 seconds

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-03-08T03:19:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2ef36a7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T03:26:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c385779
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T03:26:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2ef36a7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T03:30:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b92e9c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T03:31:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 07b03c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T03:33:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2ef36a7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-03-08T03:55:18Z

Codecov Report

Merging #2976 (db0607f) into master (dfc6eee) will decrease coverage by 0.05%.
The diff coverage is 94.28%.

@@            Coverage Diff             @@
##           master    #2976      +/-   ##
==========================================
- Coverage   89.89%   89.83%   -0.06%     
==========================================
  Files        1028      196     -832     
  Lines       83688    16596   -67092     
==========================================
- Hits        75228    14909   -60319     
+ Misses       8460     1687    -6773

Impacted Files	Coverage Δ
src/sagemaker/huggingface/model.py	`79.10% <83.33%> (+0.41%)`	⬆️
src/sagemaker/image_uris.py	`98.44% <96.55%> (-0.36%)`	⬇️
...emaker/async_inference/async_inference_response.py
...ite-packages/sagemaker/async_inference/__init__.py
...hon3.6/site-packages/sagemaker/serverless/model.py
...ython3.8/site-packages/sagemaker/local/entities.py
...ython3.7/site-packages/sagemaker/spark/__init__.py
...ib/python3.8/site-packages/sagemaker/amazon/knn.py
...n3.7/site-packages/sagemaker/workflow/functions.py
...li/compatibility/v2/modifiers/deprecated_params.py
... and 826 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfc6eee...db0607f. Read the comment docs.

sagemaker-bot · 2022-03-08T04:05:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 07b03c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:06:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c385779
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:06:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b92e9c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:17:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c385779
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:26:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b92e9c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:26:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 07b03c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:30:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b92e9c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:30:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c385779
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T04:30:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 07b03c5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid

Thank you @jeniyat that's great.🤗 I did also some testing a noticed a few things we should attack.

it is possible to use image_uris.retrieve for not supported py_version.

import sagemaker, boto3
sagemaker.image_uris.retrieve(
    framework="huggingface",
    region=boto3.Session().region_name,
    version="4.12.3",
    py_version="py38",
    base_framework_version="pytorch1.9.1",
    instance_type="neuron",
    image_scope="inference"
)

return '763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference-neuron:1.9.1-transformers4.12.3-neuron-py38-sdk1.17.1-ubuntu18.04-v1.0' but this image doesn't exist. That's quite critical since the current py_version for the default pytorch image is py38 so it's very likely that customers who want to switch from regular to inferentia might make a mistake and run into an error when deploying.

Would it make sense for solving this as well as the comment with upgrading the neuron SDK to create a new huggingface_neuron.json with a similar structure and use this to create the image uri?

In addition to this i tested to use the HuggingFaceModel to deploy.

from sagemaker.huggingface import HuggingFaceModel
import sagemaker
import boto3
sess = sagemaker.Session()

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    model_data="s3://hf-sagemaker-inference/inferentia/model.tar.gz",
	transformers_version='4.12',
	pytorch_version='1.9',
	py_version='py37',
	role=role, 
    sagemaker_session=sess
)

# Let SageMaker know that we've already compiled the model via neuron-cc
huggingface_model._is_compiled_model = True

running huggingface_model.prepare_container_def("ml.inf.xlarge") returns the following error

ValueError: Unsupported processor: inf. You may need to upgrade your SDK version (pip install -U sagemaker) for newer processors. Supported processor(s): gpu, cpu, neuron.

running predictor = huggingface_model.deploy(initial_instance_count=1, instance_type="ml.inf1.xlarge")
it returns

TypeError: expected string or bytes-like object

I guess it is coming through huggingface_model._is_compiled_model = True, but i defined it since the model was compiled outside of Sagemaker.
When i set it to false

huggingface_model._is_compiled_model = False

predictor = huggingface_model.deploy(initial_instance_count=1, instance_type="ml.inf1.xlarge")

I get the same error as for huggingface_model.prepare_container_def("ml.inf.xlarge")

Your model is not compiled. Please compile your model before using Inferentia.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-26-28b5559c63e4> in <module>
      1 huggingface_model._is_compiled_model = False
      2 
----> 3 predictor = huggingface_model.deploy(initial_instance_count=1, instance_type="ml.inf1.xlarge")

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, async_inference_config, serverless_inference_config, **kwargs)
    971                 self._base_name = "-".join((self._base_name, compiled_model_suffix))
    972 
--> 973         self._create_sagemaker_model(instance_type, accelerator_type, tags)
    974 
    975         serverless_inference_config_dict = (

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in _create_sagemaker_model(self, instance_type, accelerator_type, tags)
    516                 /api/latest/reference/services/sagemaker.html#SageMaker.Client.add_tags
    517         """
--> 518         container_def = self.prepare_container_def(instance_type, accelerator_type=accelerator_type)
    519 
    520         self._ensure_base_name_if_needed(container_def["Image"])

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/huggingface/model.py in prepare_container_def(self, instance_type, accelerator_type)
    268             region_name = self.sagemaker_session.boto_session.region_name
    269             deploy_image = self.serving_image_uri(
--> 270                 region_name, instance_type, accelerator_type=accelerator_type
    271             )
    272 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/huggingface/model.py in serving_image_uri(self, region_name, instance_type, accelerator_type)
    311             accelerator_type=accelerator_type,
    312             image_scope="inference",
--> 313             base_framework_version=base_framework_version,
    314         )

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/image_uris.py in retrieve(framework, region, version, py_version, instance_type, accelerator_type, image_scope, container_version, distribution, base_framework_version, training_compiler_config, model_id, model_version, tolerate_vulnerable_model, tolerate_deprecated_model)
    151 
    152     processor = _processor(
--> 153         instance_type, config.get("processors") or version_config.get("processors")
    154     )
    155 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/image_uris.py in _processor(instance_type, available_processors)
    338             )
    339 
--> 340     _validate_arg(processor, available_processors, "processor")
    341     return processor
    342 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/image_uris.py in _validate_arg(arg, available_options, arg_name)
    388             "Unsupported {arg_name}: {arg}. You may need to upgrade your SDK version "
    389             "(pip install -U sagemaker) for newer {arg_name}s. Supported {arg_name}(s): "
--> 390             "{options}.".format(arg_name=arg_name, arg=arg, options=", ".join(available_options))
    391         )
    392 

ValueError: Unsupported processor: inf. You may need to upgrade your SDK version (pip install -U sagemaker) for newer processors. Supported processor(s): gpu, cpu, neuron.

philschmid · 2022-03-08T08:00:24Z

src/sagemaker/image_uri_config/huggingface.json

@@ -913,7 +913,7 @@
                        "us-west-2": "763104351884"
                    },
                    "repository": "huggingface-pytorch-inference",
-                    "container_version": {"gpu": "cu111-ubuntu20.04", "cpu": "ubuntu20.04" }
+                    "container_version": {"gpu": "cu111-ubuntu20.04", "cpu": "ubuntu20.04", "neuron": "sdk1.17.1-ubuntu18.04-v1.0" }


How would this work if release a new DLC for neuron SDK 1.18.0 for the same framework version?

If sdk version is not passed as input argument, then it the retrieve will pick the latest sdk version

src/sagemaker/image_uri_config/huggingface.json

ahsan-z-khan

I would add a different json config file altogether for huggingface-neuron rather than bloating vanilla huggingface one

sagemaker-bot · 2022-03-08T21:12:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: baccba5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T21:17:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3957cac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T21:49:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: baccba5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T21:56:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3957cac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T22:18:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 3957cac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T22:27:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6ca1c71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T23:04:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: baccba5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T23:04:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 3957cac
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T23:19:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6ca1c71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T23:31:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6ca1c71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-08T23:46:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6ca1c71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid

I tried to test the updates again using the HuggingFaceModel, which now creates the following error

ValueError: Unsupported Python version: py37. You may need to upgrade your SDK version (pip install -U sagemaker) for newer Python versions. Supported Python version(s): py38.

philschmid · 2022-03-09T08:26:09Z

src/sagemaker/image_uri_config/huggingface-neuron.json

+                            "us-west-2": "763104351884"
+                    },
+                    "container_version": {"neuron": "ubuntu18.04"},
+                    "repo_versions": ["v1.0"],


Does this refer to -v1.0 in 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference-neuron:1.9.1-transformers4.12.3-neuron-py37-sdk1.17.1-ubuntu18.04-v1.0? if so we should remove this. The Image has multiple available tags see: https://github.com/aws/deep-learning-containers/releases/tag/v1.0-hf-4.12.3-pt-1.9.1-inf-neuron-sdk1.17.1-py37 one of them is 1.9.1-transformers4.12.3-neuron-py37-sdk1.17.1-ubuntu18.04 (just without the v1.0).
The v1.0 indicates the version of the container and the assimov team might update the container to apply security patches then we would need update this as well its a better practice to always use the latest version through 1.9.1-transformers4.12.3-neuron-py37-sdk1.17.1-ubuntu18.04 tag and remove repo_version

philschmid · 2022-03-09T08:26:24Z

src/sagemaker/image_uris.py

@@ -47,6 +46,8 @@ def retrieve(
    model_version=None,
    tolerate_vulnerable_model=False,
    tolerate_deprecated_model=False,
+    sdk_version=None,
+    repo_version=None,


This can then be removed

philschmid · 2022-03-09T08:26:53Z

src/sagemaker/image_uris.py

+            if not repo_version:
+                repo_version = _get_latest_versions(version_config["repo_versions"])
+            container_version = sdk_version + "-" + container_version + "-" + repo_version


sagemaker-bot · 2022-03-14T07:05:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fe4bdd3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid · 2022-03-14T08:22:31Z

While doing some more tests i noticed that you need to provide the sagemaker_session in the HuggingFaceModel, which is not the case for non-inferentia deployments.

otherwise, you ll get

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-b74825f4782f> in <module>
     14 predictor = huggingface_model.deploy(
     15     initial_instance_count=1,      # number of instances
---> 16     instance_type="ml.inf1.xlarge" # AWS Inferentia Instance
     17 )
     18 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/huggingface/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, async_inference_config, serverless_inference_config, **kwargs)
    265         if instance_type.startswith("ml.inf") and not self.image_uri:
    266             self.image_uri = self.serving_image_uri(
--> 267                 region_name=self.sagemaker_session.boto_session.region_name,
    268                 instance_type=instance_type,
    269             )

AttributeError: 'NoneType' object has no attribute 'boto_session'

according to the documentation, sagemaker_session is optional

sagemaker_session (sagemaker.session.Session) – A SageMaker Session object, used for SageMaker interactions (default: None). If not specified, one is created using the default AWS configuration chain.

sagemaker-bot · 2022-03-14T17:48:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 393fa75
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T18:33:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 393fa75
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T18:38:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 393fa75
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jeniyat · 2022-03-14T22:39:16Z

While doing some more tests i noticed that you need to provide the sagemaker_session in the HuggingFaceModel, which is not the case for non-inferentia deployments.

otherwise, you ll get

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-b74825f4782f> in <module>
     14 predictor = huggingface_model.deploy(
     15     initial_instance_count=1,      # number of instances
---> 16     instance_type="ml.inf1.xlarge" # AWS Inferentia Instance
     17 )
     18 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/huggingface/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, async_inference_config, serverless_inference_config, **kwargs)
    265         if instance_type.startswith("ml.inf") and not self.image_uri:
    266             self.image_uri = self.serving_image_uri(
--> 267                 region_name=self.sagemaker_session.boto_session.region_name,
    268                 instance_type=instance_type,
    269             )

AttributeError: 'NoneType' object has no attribute 'boto_session'

according to the documentation, sagemaker_session is optional

sagemaker_session (sagemaker.session.Session) – A SageMaker Session object, used for SageMaker interactions (default: None). If not specified, one is created using the default AWS configuration chain.

Handled this case in latest revision

src/sagemaker/image_uris.py

sagemaker-bot · 2022-03-14T23:09:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d993da0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T23:09:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 02445ba
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T23:45:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d993da0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-14T23:45:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 02445ba
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-15T00:04:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7210d2f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-15T00:04:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 33adabc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-15T00:24:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: db0607f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-15T01:20:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: db0607f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-15T02:07:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: db0607f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

philschmid reviewed Mar 8, 2022

View reviewed changes

ahsan-z-khan suggested changes Mar 8, 2022

View reviewed changes

philschmid reviewed Mar 9, 2022

View reviewed changes

add session value for HF model

393fa75

jeniyat requested review from navinsoni and ahsan-z-khan March 14, 2022 22:36

navinsoni reviewed Mar 14, 2022

View reviewed changes

src/sagemaker/image_uris.py Outdated Show resolved Hide resolved

update w/ fstring

d993da0

navinsoni reviewed Mar 14, 2022

View reviewed changes

src/sagemaker/image_uris.py Outdated Show resolved Hide resolved

update w/ fstring

02445ba

navinsoni approved these changes Mar 14, 2022

View reviewed changes

jeniyat added 2 commits March 14, 2022 17:53

added to do

7210d2f

added to do

33adabc

ahsan-z-khan approved these changes Mar 14, 2022

View reviewed changes

update comment

db0607f

jeniyat merged commit 7e2c7ab into aws:master Mar 15, 2022

jeniyat deleted the jeniyat/hf-inf-neuron branch March 15, 2022 03:01

jerrypeng7773 pushed a commit to jerrypeng7773/sagemaker-python-sdk that referenced this pull request May 13, 2022

feature: Inferentia Neuron support for HuggingFace (aws#2976)

588dd69

feature: Inferentia Neuron support for HuggingFace #2976

feature: Inferentia Neuron support for HuggingFace #2976

Conversation

jeniyat commented Mar 8, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

codecov-commenter commented Mar 8, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

philschmid left a comment

Choose a reason for hiding this comment

philschmid Mar 8, 2022

Choose a reason for hiding this comment

jeniyat Mar 14, 2022

Choose a reason for hiding this comment

ahsan-z-khan left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 8, 2022

AWS CodeBuild CI Report

philschmid left a comment

Choose a reason for hiding this comment

philschmid Mar 9, 2022

Choose a reason for hiding this comment

philschmid Mar 9, 2022

Choose a reason for hiding this comment

philschmid Mar 9, 2022

Choose a reason for hiding this comment

sagemaker-bot commented Mar 14, 2022

AWS CodeBuild CI Report

philschmid commented Mar 14, 2022

jeniyat commented Mar 8, 2022 •

edited

Loading

codecov-commenter commented Mar 8, 2022 •

edited

Loading